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Abstract 

In  spite  of  the  surge  in  new  technologies,  complex  operational  processes,  and  high 
operations  tempos,  the  selection,  education,  training,  and  team  functions  of  the  modern 
C2  operators  has  not  been  given  comparable  attention.  In  order  to  properly  staff  future 
C2  operations  centers  with  capable  individuals  and  operational  teams,  attention  needs  to 
be  directed  to  the  analysis  of  the  evolving  C2  operator  training  requirements  and  team 
dynamics  that  are  being  driven  by  modern  technology  and  changing  concepts  of 
operations. 

This  paper  will  describe  the  assessment  approach  and  methodology  employed  by 
the  Air  Force  Research  Laboratory’s  Warfighter  Training  Research  Division,  Information 
Systems  Training  Branch,  Brooks  AFB,  TX  (AFRL/HEAI)  researchers  as  they  observed 
the  C2  operators  in  the  Time  Critical  Targeting  Cell  (TCTC)  during  Joint  Expeditionary 
Force  experiment  2000  (JEFX  00).  The  AFRL/HEAI  researchers  followed  a  human- 
factors  oriented  approach  in  developing  a  data  collection  methodology  to  gain  insight  into 
the  scope  of  issues  confronting  personnel  in  modern  C2  operation  centers.  This  approach 
stressed  the  importance  of  C2  functions  and  processes  related  to  time  critical  targeting 
that  depended  on  the  interactions  among  individuals  and  teams  of  C2  operators.  These 
insights  may  ultimately  assist  in  the  specification  of  C2  operator  training  requirements 
and  help  characterize  the  individual  and  team  “desired”  performance  capabilities. 


SECTION  1.0 
INTRODUCTION 


1.1  Background 

As  a  result  of  similar  interests  in  conducting  research  and  experimentation  with 
Air  Force  Command  and  Control  (C2)  processes,  AFRL/HEAI  volunteered  the  services 
of  several  of  its  research  staff  to  assist  in  the  assessment  of  C2  issues  during  the  JEFX 
00JEFX  00  evolutionary  activities.  The  JEFX  00  activities  the  AFRL/HEA  team 
members  participated  in  included  three  spiral  development  events  and  the  main  JEFX  00 
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experiment.  The  JEFX  00  events  occurred  during  various  periods  between  15  March  and 
15  September  2000.  Assessments  of  the  TCTC  Process  Initiative  were  accomplished  by 
the  team  members  during  Spiral  2,  3  and  the  Main  Experiment  at  Nellis  AFB,  Nevada. 
The  AFRL/HEAI  analysts,  as  members  of  the  AFEO  Assessment  Team,  were  tasked  to 
collect  and  provide  information  on  organizational  processes  and  interfaces  and  to 
examine  and  provide  descriptive  measures  from  a  human  factors  perspective  in  the  areas 
of  communications  patterns,  personnel,  and  team  work.  In  addition,  the  team  was  asked 
to  look  at  organization  and  leadership  from  the  Doctrine,  Organization,  Training, 
Materiel,  Leadership,  and  Personnel  (DOTMLP)  perspective  and  provide  this  information 
in  a  form  most  useful  to  the  warfighter. 


1.2  Scope 

Due  to  the  high  visibility  of  the  JEFX  00  experiment,  the  competing  objectives  of 
numerous  Air  Force  Agencies,  and  the  associated  environmental  constraints,  only  three 
research  analysts  were  permitted  to  take  part  in  each  of  the  spiral  and  Main  Experiment 
events.  The  research  analysts  designed  measures  of  merit  and  data  collection  forms  of 
various  formats  to  garner  raw  operational  data  and  record  individual  observations.  The 
analysts  spent  their  entire  time  collecting  information  and  recording  the  comments  and 
ideas  expressed  by  TCT  Cell  members  for  more  than  35  days  of  activities  encompassing 
the  three  spirals  and  the  high  intensity  period  of  the  Main  Experiment. 


1.3  Experiment  Overview 

The  Joint  Expeditionary  Force  experiments  provide  the  Air  Force  with  a  vehicle 
for  experimentation  with  operational  concepts  and  technologies  that  will  enhance 
aerospace  capabilities  in  the  21st  century.  JEFX  00  is  the  third  in  a  series  of  large-scale 
Air  Force  experiments.  Previous  Joint  Expeditionary  Force  experiments  were 
conducted  in  1998  and  1999.  EFX  ’98  was  the  Air  Force’s  first  large-scale  experiment 
and  explored  the  ability  to  move  information  while  deploying  fewer  people  and  less 
equipment.  JEFX  ’99  built  upon  the  lessons  learned  in  the  ’98  experiment  to  better 
integrate  space  personnel  and  resources  into  the  expeditionary  aerospace  operations. 
JEFX  00  focused  on  agile  combat  support  and  time-critical  targeting  that  will  provide  an 
enhanced  capability  to  the  Joint  Force  and  the  Combined  Forces  Air  Component 
Commander  (CFACC). 

JEFX  is  not  an  exercise,  test,  or  technology  demonstration.  The  JEFX  process  is 
a  discovery  process  that  integrates  new  and  emerging  technologies,  solves  deficiencies, 
and  provides  advanced  capabilities.  The  main  point  of  these  experiments  is  to  learn  and 
discover  new  and  improved  methods  and  processes  for  conducting  military  operations, 
not  to  train.  These  experiments  are  designed  to  involve  the  warfighters  early  in  the 
development  of  new  concepts  and  technologies.  This  process  allows  the  Air  Force  to 
provide  more  operationally  focused  systems  and  processes  necessary  to  reach  Joint 
Vision  2020. 
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The  main  objectives  for  JEFX  00  were  to  explore  improvements  in  global 
mobility;  intelligence;  surveillance  and  reconnaissance  battle  management;  and  a  new 
concept  that  allows  data  to  be  turned  into  information  for  our  warfighters  -  the  Joint 
Battlespace  Infosphere.  To  this  end,  JEFX  00  examined  45  processes  and  system 
initiatives  with  a  supporting  cast  of  more  than  3,000  military  participants  at  11  sites 
across  the  nation.  JEFX  00  combined  the  use  of  models,  simulations,  and  live-fly 
operations. 

The  three  main  operating  locations  for  JEFX  00  were  Langley  AFB,  VA,  Hurlburt 
Field,  FL,  and  Nellis  AFB,  NV.  Located  at  Langley  AFB  was  the  Operations  Support 
Center  (OSC)  where  many  of  the  “reach-back”  and  agile  combat  support  functions  were 
explored.  The  Combined  Air  Operations  Center-Rear  (CAOC-R)  was  located  at  Hurlburt 
Field.  The  CAOC-R  was  the  node  from  which  the  CFACC  developed  and  promulgated 
theater  objectives.  Located  at  Nellis  AFB  was  a  secured  compound  that  was  designated 
the  Combined  Air  Operations  Center-Forward  (CAOC-F)  which  contained  a  Time 
Critical  Targeting  Cell  (TCTC).  A  macro  view  of  the  participant  nodes  and  organizations 
is  depicted  in  Figure  1-1.  The  AFRL/HEAI  personnel  were  located  at  Nellis  AFB  and 
served  inside  the  TCTC  as  part  of  the  AFEO  Assessment  Team.  The  scope  of  the 
AFRL/HEAI  JEFX  00  assessment  is  limited  to  the  TCTC  process  initiative  in  the  CAOC- 
F  conducted  at  Nellis  AFB. 
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Figure  1-1  The  “Big  Picture”  of  JEFX  00  influences  and  nodes 
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Prior  to  the  Main  Experiment,  there  were  many  days  of  spiral  rehearsals  to  work 
out  the  integration  of  systems  and  the  operational  procedures  needed  to  accomplish  the 
Time  Critical  Targeting  mission.  Each  spiral  provided  analysts  the  opportunity  to  see  the 
evolution  of  operator  procedures,  team  processes,  and  gain  experience  with  prototype 
systems.  The  final  experiment  consisted  of  additional  operator  training,  process  and 
technology  practice,  simulation,  and  live-fly  periods. 

During  the  Main  Experiment,  virtual  and  constructive  simulation  and  live-flying 
operations  were  employed.  The  TCT  Cell  at  Nellis  AFB,  Nevada  was  part  of  the 
Combined  Air  Operations  Center  (CAOC)  Forward  with  the  CAOC-Rear  at  Hurlburt 
Field,  FL  and  the  Operations  Support  Center  (OSC)  located  at  Fangley  AFB,  VA. 

A  scenario  that  allowed  for  time-critical  targets  to  “emerge”  drove  each  day  of  the 
experiment.  The  targets  had  to  be  detected  using  ISR  assets,  prioritized,  coordinated  with 
the  CAOC  for  a  destruction  decision,  and  committed  on  by  the  appropriate  and  available 
weapon  assets.  The  air  battle  management  section  controlled  the  assigned  attack 
platform(s),  and  the  target  was  assessed  for  battle  damage  through  aircrew  reports  and 
tactical  and  national  ISR  assets.  All  of  these  activities  were  accomplished  by  the  TCT 
cell,  which  consisted  of  a  Command  Section,  Hunter  Section,  Killer  Section,  datalink 
Interface  Control  Section,  and  Joint  Battlespace  Infosphere  Management  Section.  The 
mission  of  the  assessment  team  was  to  observe  these  teams  in  action  and  derive  as  much 
information  about  their  activities  and  processes  as  possible. 


1.4  Assessment  Participants  at  Nellis  AFB 

Government  military,  civilian,  and  contractor  personnel  from  around  the  country 
supported  the  assessment  of  the  technology  and  processes  evaluated  in  the  TCTC  at 
Nellis  AFB.  In  addition  to  the  three  AFRF/HEAI  assessment  personnel,  there  were 
numerous  contractors  from  multiple  companies  who  were  assigned  to  the  TCTC 
Assessment  Team.  The  Nellis  AFB  TCTC  Assessment  Team  included  representatives 
from  Air  Combat  Command  (ACC),  the  Aerospace  Command  and  Control,  Intelligence, 
Surveillance  &  Reconnaissance  Center  (AC2ISRC),  the  Command  and  Control  Training 
and  Innovation  Group  (C2TIG),  1st  Air  Force,  and  the  552  Air  Control  Wing. 

The  AFRF/HEAI  Command  and  Control  Training  Research  (C2TR)  assessment 
team  was  able  to  orchestrate  having  a  number  of  its  members  attend  portions  of  the 
various  JEFX  00  events.  At  an  assortment  of  times  during  the  three  Spirals  and  the  Main 
Experiment,  the  C2TR  personnel  listed  in  Figure  1-2  had  the  opportunity  to  attend  the 
specified  JEFX  00  activities. 
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Name 

Organization 

Activities  Attended 

Capt.  Ed  McCormick 

AFRL/HEA  C2TR  Program  Manager 

Spiral  1,  2,  and  3 

Mike  Garrambone 

Veridian  Engineering 

Spiral  3,  ME 

Matt  Dalrymple 

Veridian  Engineering 

Spiral  2 

Debbie  Hall 

Veridian  Engineering 

Spiral  3 

Mike  Goodman 

Veridian  Engineering 

Spiral  1,  2,  3,  ME 

Mike  Paley 

Aptima  Inc. 

Spiral  1,  2,  ME 

Sarah  Miescher 

Aptima  Inc. 

Spiral  1,  3 

Spiral  1  -  C2  Unified  Battlespace  Environment  (CUBE),  Hanscom  AFB,  15-22  Mar  ’00 

Spiral  2  -  Nellis  AFB,  30  May  -  9  Jun  ’00 

Spiral  3  -  Nellis  AFB,  3-13  Aug  ‘00 

Main  Experiment  (ME)  -  Nellis  AFB,  28  Aug  -  15  Sep  ’00 

Figure  1-2.  AFRL/HEA  C2TR  Team’s  Participants  in  JEFX  00 


1.5  Experimental  Matrix  for  JEFX  00 

Although  JEFXs  are  designed  and  referred  to  as  experiments,  they  are  actually  not 
experiments  in  the  classical,  scientific,  or  statistical  sense.  These  experiments  are  much 
too  complex  to  boil  down  to  the  investigation  of  one  or  two  variables  that  can  be 
evaluated  to  accept  or  reject  a  hypothesis.  The  JEFX  experiments  have  hypotheses,  but 
they  are  addressed  in  a  more  qualitative  vice  quantitative  manner.  The  experiments  are 
not  demonstrations.  They  have  carefully  considered  specific  measures  of  effectiveness. 
Unlike  a  demonstration,  failure  is  acceptable— in  fact,  just  as  much  can  be  learned  from 
failure  as  from  success.  Consequently,  the  JEFX  format  is  all  about  experimenting  with 
new  and  innovative  technologies  and  processes  to  discover  improved  ways  to  conduct 
military  operations. 

Because  JEFX  00  was  not  a  laboratory-controlled  experiment  in  the  classical 
sense,  a  standard  statistical  experimental  design  did  not  exist.  There  were,  however, 
measures  of  effectiveness  and  performance  applicable  to  desired  assessment  outcomes. 

The  JEFX  00  Initiatives  were  assessed  from  five  perspectives  which  included 
technical,  warfighter  utility,  thread  assessment,  joint  capabilities  level,  and 
experimentation  management.  The  technical  assessment  was  completed  by  the 
Electronic  Security  Center  prior  to  JEFX  to  ensure  each  initiative  technically  worked. 
The  warfighter  utility  assessed  the  utility  of  the  initiative  in  performing  operational  tasks. 
The  thread  assessment  incorporated  the  evaluation  of  how  well  multiple  initiatives 
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worked  together  in  support  of  a  specific  area  of  responsibility.  The  joint  capabilities 
level  assessment  involved  determining  whether  the  initiatives  have  Joint  applications  and 
did  it  aid  the  JFACC/JTF  in  accomplishing  Joint  tasks.  Experiment  management  was 
also  evaluated  to  assess  whether  improvements  could  be  made  in  planning,  designing, 
executing,  and  assessing  the  JEFX  experiments. 


SECTION  2.0 

AFRL/HEAI’S  INTEREST  IN  JEFX  00 
2.1  AFRL/HEAI  APPROACH 

The  Information  Systems  Training  Research  Branch,  as  a  focused  research  arm  of 
the  Air  Force  Research  Laboratory,  has  a  vested  interest  in  helping  to  assess  and 
characterize  the  multitude  of  activities  that  collectively  comprise  the  Air  Force  command 
and  control  (C2)  process.  There  is  currently  a  revolutionary  atmosphere  that  is  driving  a 
comprehensive  review  of  the  way  the  Air  Force  has  historically  conducted  C2  operations. 
In  keeping  pace  with  this  revolution,  the  JEFX  experimentation  test  bed  has  created  an 
excellent  venue  for  assessing  not  only  the  new  technology  of  the  future,  but  also  the  role 
of  the  human  element  in  the  facilitation  and  management  of  the  overall  C2  processes. 

Twenty-first  century  warfare  has  mandated  the  need  to  integrate  orders  of 
magnitude  more  information  sources  provided  by  new  sensors  to  develop  a  coherent 
picture  of  the  battlespace  and  to  formulate  C2  responses  in  a  distributed  environment. 
Today,  C2  operators  are  required  to  manage  larger  and  larger  quantities  of  battlespace 
information,  and  to  do  it  more  quickly  in  situations  that  are  unlike  those  of  former  ages. 
An  assessment  of  the  role  of  the  human  element  within  this  new  and  evolving  C2 
environment  is  needed  to  help  determine  the  selection  of  the  optimum  type  of  individuals 
to  serve  in  these  C2  operation  centers.  Equally  important  is  the  evaluation  of  the 
changing  role  of  team  functions/processes,  and  the  implications  of  operator  workload 
levels  including  psychomotor  and  cognitive  loads,  etc.  AFRL/HEAI,  therefore,  proposed 
accomplishing  a  human  factors-oriented  assessment  of  the  human  element  within  and 
throughout  the  TCTC  experimentation  processes  as  part  of  the  JEFX  Assessment 
Integrated  Process  Team. 

The  primary  purpose  for  AFRL/HEAI’ s  interest  in  the  JEFX  00  environment  was 
to  gain  insight  into  the  scope  of  modern  issues  that  are  driving  the  C2  operator  training 
requirements.  AFRL/HEAI  CTR’s  goals  are  to  provide  the  best  technology,  methods,  and 
processes  to  train  the  modern  warfighters.  The  C2TR  team  members  are  concentrating 
on  the  C2  operator  training  needs  and  the  best  means  to  train  them  faster,  better,  and 
cheaper. 

The  data  for  this  assessment  was  gained  through  observation  of  the  JEFX  00  C2 
operator  teams  in  the  performance  of  their  duties  and  through  consultation  with  operators, 
as  appropriate,  on  a  non-interference  basis.  All  output  from  the  AFRL/HEAI  assessment 
was  and  will  continue  to  be  coordinated  through  the  Air  Force  Experimentation  Office. 
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The  AFRL/HEAI  assessment  team  was  comprised  of  three  C2TR  analysts  who  possess 
human  factors  and  C2  subject  matter  experience  and  expertise. 

The  AFRL/HEAI  personnel  represented  the  training-research  component  of  the 
assessment  team.  Although  the  first  obligation  was  to  support  the  assessment 
requirements  of  the  AFEO  Assessment  Team,  the  AFRL/HEA  representatives  brought  a 
different  perspective  to  evaluation  of  the  activity  occurring  in  the  TCTC  at  Nellis  AFB. 
The  primary  interests  of  the  AFRL/HEAI  researchers  included  the  communication  and 
information  flows,  the  team  building  and  interaction  activities,  coordination  processes, 
information  management,  leadership,  operator  training  and  other  human  factor  issues. 

The  analysts  integrated  with  and  supported  the  Assessment  Branch  to  assist  in 
assessment  of  the  Time  Critical  Targeting  Cell  Process  Initiative  conducted  at  Nellis 
AFB.  Although  the  primary  AFRL/HEA  “going-in”  focus  was  on  team  interactions, 
training,  and  communications  patterns,  AFEO  requested  that  the  AFRL/HEA  team  look 
at  drawing  some  conclusions  on  leadership  and  people  without  “evaluating”  individual  or 
group  performance. 


2.2  133rd  Air  Control  Squadron 

Previous  work  with  the  133rd  Air  Control  Squadron  (133rd  ACS)  Ft.  Dodge,  IA 
was  a  contributing  factor  that  influenced  AFRL/HEAI’ s  desire  to  participate  in  JEFX  00. 
The  133rd  ACS,  as  a  Control  and  Reporting  Element  (CRE)  of  the  ground-based  Theater 
Air  Control  System  (GTACS),  had  been  helpful  in  providing  AFRL/HEAI  with  insight 
into  the  roles,  functions,  and  missions  of  the  GTACS  C2  operators.  The  surveillance, 
weapons,  and  battle  management  C2  operators  from  the  133rd  ACS  had  provided  a  lot  of 
useful  information  to  AFRL/HEAI  analysts  investigating  the  operator  training  challenges 
in  these  types  of  C2  organizations.  However,  AFRL/HEAI’ s  desired  research  scope 
encompassed  not  only  the  in-garrison  CRE  C2  operator  functions  but  also  those 
additional  functions  performed  when  connected  to  a  netted  theater  architecture.  In 
addition,  AFRL/HEAI’ s  goals  included  research  on  the  C2  operator  training  requirements 
from  all  theater  C2  nodes. 

In  light  of  AFRL/HEAI’ s  desires  to  observe  other  C2  nodes  as  well  as  the  133rd 
ACS  in  a  netted  environment,  the  133rd  ACS  encouraged  the  AFRL/HEAI’s  C2TR  team 
to  observe  them  during  JEFX  00.  The  purpose  of  the  observation  would  be  to  gain 
additional  insight  and  to  also  assist  the  133rd  ACS  by  providing  them  with  an  objective 
and  credible  assessment  of  their  performance  during  time-critical  targeting  operations. 
Because  the  133rd  ACS  provided  the  strike  control  and  overall  target  execution  functions 
in  the  TCTC  during  JEFX  00,  observation  of  their  varied  C2  tasks  was  deemed  by  C2TR 
analysts  as  a  desirable  activity  to  pursue.  As  a  direct  result  of  the  invitation  by  the  133rd 
ACS,  the  C2TR  team  sought  appropriate  avenues  for  integrating  into  the  JEFX  00  TCTC 
operation. 
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SECTION  3.0 

ASSESSMENT  METHODOLGY 


3.1  Human  Factors  Considerations 

The  purpose  of  this  section  of  the  report  is  to  document  the  human  factors 
considerations  used  to  develop  the  data  collection  methodology,  which  ultimately 
influences  the  analysis,  interpretation,  and  conclusions  from  this  exercise.  In  the  interests 
of  brevity,  some  brief  background  will  be  provided  but  the  emphasis  will  be  on  the  major 
issues. 


The  time  critical  targeting  process  is  being  enhanced  through  the  parallel  efforts 
of  technology  application  and  human  operator  methodology.  This  approach  yields 
greater  synergism  than  an  approach  that  focuses  exclusively  on  hardware/software  or 
crew  organization/training  individually.  The  parallel  approach  is  driven  by  the  nature  of 
the  changing  operational  environment  for  time  critical  targets  including: 

1.  Vastly  increased  quantities  of  data  and  information 
including  imagery  from  localized  and  national  assets. 

2.  Reduced  time  available  for  prosecuting  time  critical 
targets  such  as  Scuds/TELs,  SA-10C,  SA-15,  etc. 

3.  Evolving  roles  for  individuals  and  teams  within  the  C2 
decision  structure. 

Based  upon  the  above  considerations,  Veridian  Engineering  supported 
AFRL/HEAI  with  developing  a  crew-centered  approach  for  conducting  an  assessment 
during  JEFX  00.  This  approach  stressed  the  importance  of  C2  functions  and  processes 
related  to  time  critical  targeting  that  depend  on  interactions  among  individuals  and  teams. 
As  a  result,  the  most  significant  issues  that  were  addressed  included  workload,  cognitive 
load,  individual  and  team  processes,  information  flow,  and  decision  making.  The  intent 
was  to  use  JEFX  00  to  obtain  insights  for  shaping  efforts  to  (1)  optimize  the  architecture 
of  various  C2  organizations,  and  (2)  develop  training  research  facilities  and 
methodologies  used  to  enhance  crew  performance.  The  context  of  data  collection 
influenced  the  allowable  methodology  options  as  well  as  the  interpretation  and 
conclusions  that  were  drawn  from  the  data  that  was  obtained  from  this  experiment. 

The  concepts  that  form  the  basis  for  Figure  3-1  can  be  used  to  envision  some 
generalized  relationships  among  the  data  collection  context  and  several  factors  associated 
with  data  capture,  data  reduction,  data  analysis,  and  formulating  conclusions.  To  use  the 
figure,  one  can  select  a  “context  region”  that  represents  the  approximate  operating  region 
for  data  collection.  The  center  of  the  figure  relates  the  general  context  breadth  over 
which  studies  can  be  conducted.  For  any  given  context  selected,  one  can  draw  an 
imaginary  line  both  above  and  below  this  region  to  determine  various  implications  for  the 
study  based  upon  the  context.  Although  not  shown  in  the  figure,  JEFX  00  was 
envisioned  as  being  between  the  “middle  ground”  and  the  right  hand  side  of  the  figure. 


In  other  words,  the  context  was  one  of  a  field  exercise  where  crew  decision  making  has 
been  labeled  as  “naturalistic”  by  some  authors;  this  means  that  highly  trained  operators 
used  actual  hardware  and  software  for  realistic  decision  making  according  to  a  combat- 
like  event  flow  for  time  critical  targets.  This  is  in  contrast  to  “the  left  side”  of  the  context 
region,  laboratory  environments  studying  classical  decision  making,  where  it  is  common 
to  use  college  sophomores  as  subjects,  with  simulated  hardware  and  software  used  for 
making  simplistic  decisions  with  an  artificial  event  flow.  Understanding  the  context  of 
the  study  allowed  the  project  team  to  optimize  the  data  collection  methodology  and 
understand  the  bounds  of  data  processing. 
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Figure  3-1.  Context  vs.  Science 


The  term  “fidelity”  in  the  figure  relates  primarily  to  functional  fidelity  which  has 
to  do  with  the  realism  of  the  functions  and  tasks  performed  by  the  C2  crew  and  not 
merely  physical  or  face  validity  where  the  appearance  of  the  support  hardware  is  the 
issue.  Due  to  the  nature  of  JEFX  00,  it  was  generally  agreed  that  we  were  to  operating  in 
a  fairly  realistic  context.  This  has  the  side  benefit  of  affording  high  “operational 
relevance”  because  the  what,  when,  why,  and  how  of  C2  decision  making  were  very 
much  like  combat  operations  for  time  critical  targeting.  Focusing  on  the  lower  portion  of 
Figure  3.1  one  can  assess  the  degree  of  experimental  control  implied  by  the  respective 
context.  Faboratory  environments  lend  themselves  to  very  high  degrees  of  control,  what 
factors  are  held  constant,  and  which  are  allowed  to  vary.  As  one  moves  to  the  right 
toward  field  studies,  there  is  less  control  based  upon  various  practical  constraints;  this 
starts  to  approximate  the  “fog  of  war”  where  uncontrolled  factors  can  be  quite  numerous 
and  influential.  This  lack  of  control  does  not  invalidate  the  findings  from  operational 
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contexts,  but  it  does  place  a  requirement  on  recording  factors  surrounding  the  event  flow 
so  that  one  has  some  insights  about  influences  to  the  results.  These  influences,  or 
causative  factors  as  shown  on  the  figure,  are  clearer  in  a  laboratory  context  because  so 
many  factors  are  held  constant  and  only  a  limited  number  are  varied  over  a  given  range. 
The  reverse  is  true  in  field  contexts  because  it  is  not  possible  to  precisely  control  large 
numbers  of  factors  to  the  degree  that  one  can  in  a  laboratory  setting.  Hence,  the  more 
factors  that  are  allowed  to  vary,  the  more  competing  causes  exist  for  explaining  or 
confounding  the  results.  These  competing  causes  may  exert  influence  individually  or 
may  interact  with  each  other  in  complex  ways. 

Although  not  shown  in  the  figure,  there  are  additional  factors  that  are  strongly 
related  to  the  choice  of  context  and  include  number  of  crewmembers,  number  of 
replications,  and  measures  of  effectiveness.  On  one  end  of  the  spectrum,  for  laboratory 
studies,  it  is  often  common  to  have  from  20  to  100  or  more  subjects  or  teams  based  upon 
such  practical  factors  as  subject  availability,  scheduled  time  for  data  collection,  and 
reimbursement  costs.  However,  at  the  other  end  of  the  spectrum,  for  field  studies,  the 
practicalities  of  schedules  for  range  time,  availability  of  information  and  weapon 
systems,  crew  training,  security,  et  cetera,  often  dictate  that  small  numbers  of  replications 
for  any  given  treatment  condition  of  interest.  Quite  a  few  more  replications  are  practical 
for  laboratory  settings  and  are  often  pursued  to  obtain  stable  data  from  a  statistical 
perspective.  In  field  exercises/experiments,  such  as  JEFX  00,  a  replication  of  “1”  is 
sometimes  all  that  is  practical.  Note  that  it  is  not  only  more  difficult  to  obtain  a  greater 
number  of  replications  in  field  settings,  it  is  also  more  difficult  to  control  the  data 
collection  environment  to  ensure  equivalency  of  treatment  replications.  There  are  too 
many  factors  that  cannot  be  controlled  in  the  field  that  are  easy  to  standardize  in 
laboratory  environments.  Examples  include  weather  and  visibility  conditions,  location  of 
participants  for  the  exercise,  unexpected  events,  and  compliance  with  rules  of 
engagement.  As  will  be  shown,  the  above  issues  strongly  relate  to  measures  of 
effectiveness. 

Laboratory  studies  typically  entail  the  use  of  several  measures  of  effectiveness 
and  some  studies  have  used  very  large  number  of  measures.  The  number  of  measures  is 
driven  not  just  by  practicalities  of  the  instrumentation  systems  used  to  collect  the  data  but 
also  by  the  nature  of  the  studies  themselves.  Exploratory  studies  will  often  measure  a 
large  number  of  (dependent)  variables  or  measures  as  part  of  an  overall  effort  to  establish 
and  understand  relationships.  Yet  in  other  cases,  there  is  a  need  to  establish  a 
performance  database  and  hence  large  numbers  of  measures  will  likewise  be  obtained. 
The  number  of  measures  in  laboratory  settings  has  relatively  few  bounds  because  there  is 
a  desire  to  learn  as  much  as  possible  in  the  interests  of  science  and  not  necessarily  what 
has  immediate  practical  application;  conclusively  identifying  causative  or  diagnostic 
factors  is  of  great  interest.  As  one  deviates  from  laboratory  environments  and  moves 
toward  field  exercises,  there  is  a  general  trend  toward  fewer  measures  and  the  measures 
tend  to  be  more  descriptive  than  diagnostic,  plus  they  shift  from  the  crew  task/function 
perspective  to  the  engagement  or  mission  level  perspective.  Field  exercises  tend  to  focus 
on  using  measures  that  have  immediate  operational  relevance  such  as  probability  of  kill, 
exchange  ratios,  or  resources  expended. 
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As  used  here,  the  term  “measures”  embodies  three  distinct  yet  related  issues:  (1) 
operational  definition  of  measures,  (2)  criteria,  and  (3)  standards.  There  is  quite  a  bit  of 
latitude  in  terms  of  how  these  issues  apply  to  the  various  data  collection  contexts,  but  in 
the  interests  of  brevity,  only  a  cursory  discussion  will  be  presented  here. 

3.2  Measures 

With  regard  to  operational  definition  of  measures,  what  is  meant  here  is  an  overt 
statement  of  what  is  going  to  be  measured,  how,  when,  and  to  what  resolution  for  each 
respective  “measure.”  In  some  instances,  there  is  also  a  need  to  define  the  processes 
associated  with  data  processing.  The  topic  of  operational  definition  of  measures  is 
particularly  critical  with  regard  to  human  operator  experimentation,  such  as  JEFX  00, 
because  there  is  so  much  misunderstanding  operator/crew  measures  and  some  measures 
do  not  exist  in  reality,  they  are  theoretical  constructs.  Unlike  system  or  mission  level 
measures  such  as  number  of  targets  prosecuted,  number  of  assets  utilized,  or  exchange 
ratios,  crew/team  measures  such  as  workload,  situation  awareness,  or  leadership  do  not 
have  “plug  and  play  meters”  that  can  be  inserted  into  the  system  for  automated  data 
collection.  Instead,  there  must  be  a  description  of  what  the  theoretical  constructs  are  in 
terms  of  what  is  measured,  how,  and  when.  For  example,  workload  might  be  defined  as 
an  AW  ACS  operator  rating  how  much  more  difficulty  it  is  to  assign  fighters  to  a  target 
compared  to  merely  monitoring  the  air  situation;  the  task  might  be  rated  as  twice  as 
difficult,  three  times  as  difficult,  and  so  forth. 

Specifying  specific  criteria  is  an  issue  that  applies  to  all  measures,  those  that  are 
crew-centered  as  well  as  others.  For  example,  a  mission  level  measure  might  be 
survivability.  An  associated  criterion  might  be  number  of  AAA  rounds  hitting  an 
aircraft.  Some  criteria  have  established  credibility  due  to  longstanding  use  in  the 
laboratory  through  field  contexts  while  others  have  been  newly  developed  and  must  be 
used  with  caution. 

Then  there  is  the  topic  of  standards  that  refers  to  an  agreed  upon  or  decreed  value 
of  a  given  criterion.  For  example  above,  a  standard  might  be  “if  5  AAA  rounds  hit  an 
aircraft,  it  is  considered  a  kill.”  Standards  are  not  as  common  for  crew-centered 
measures,  however.  Many  of  the  reasons  for  this  are  due  to  the  fact  that  quantifying  “the 
human  system”  is  vastly  more  complex  than  for  other  physical  systems.  The  relationship 
between  crew  performance  and  system  performance  is  not  always  clear  cut.  For 
example,  an  AAA  gunner  might  be  exposed  to  the  same  target  aircraft  twice  in 
succession.  One  time  the  gunner  might  rate  the  engagement  as  difficult  and  achieve  a 
low  to  moderate  probability  of  kill,  while  the  second  time  the  gunner  might  also  rate  the 
engagement  as  difficult  yet  achieve  a  high  probability  of  kill.  So  using  a  rating  of 
“difficult”  as  a  standard  is  not  as  straightforward  as  one  might  hope.  Sometimes  the 
project  team  relies  on  relative  comparisons  such  as  “system  A  is  easier  to  use  than 
systems  B  and  C”  due  to  the  lack  of  an  agreed  -upon  standard. 
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For  JEFX  00,  as  related  to  time  critical  targeting  from  the  crew  perspective,  there 
were  several  considerations  that  strongly  influenced  the  data  capture  methodology  that 
was  developed.  These  considerations  were: 

1.  JEFX  00  was  more  of  an  operational  exercise  than  a 
test  and  evaluation  study. 

2.  Complete  and  accurate  knowledge  of  the  truth  set 
might  not  be  readily  available  for  the  entire  exercise. 

3.  Crewmembers,  while  all  could  be  expected  to  be 
highly  trained,  cannot  be  individually  selected,  may 
be  rotated  during  JEFX  00,  and  could  occasionally  be 
substituted  with  replacements. 

4.  Interactions  among  crewmembers  or  with  external 
organizations  are  at  the  discretion  of  the 
crewmembers  and  are  not  under  the  control  of 
observers. 

5.  There  should  be  an  attempt  to  assess  crew-centered 
issues  at  the  task,  function,  and  mission  levels 
yielding  quantitative  and  qualitative  data. 

6.  Automated  instrumentation  is  unavailable. 

7.  While  data  capture  can  be  obtrusive,  it  must  not  be 
invasive. 


3.3  Assessment  Methodology  Tools 


The  AFRL/HEAI  assessment  team  employed  a  proven  subjective  assessment  tool 
to  quantify  information  and  communication  flow  within  an  organization  during  Spiral  3 
and  the  Main  Experiment  of  JEFX  00.  Aptima,  Inc.  had  successfully  utilized  these  same 
data  collection  methods  during  the  US  Navy’s  Global  2000  exercise  and  operator- in-the- 
loop  experiments.  These  measurement  techniques  were  designed  to  focus  on  interactions 
and  events  that  are  critical  to  organizational  effectiveness  and  provide  insight  into  how 
team  interaction  develops  over  time.  This  data  collection  was  designed  to  be  tied  to 
critical  mission  events  and  addresses  multiple  aspects  of  the  organizational  process.  As 
new  information  regarding  a  critical  event  is  injected  into  an  organization  (i.e., 
identification  of  a  new  hostile  threat),  specific  data  is  collected  to  assess  how  this 
information  is  managed  and  transmitted  throughout  the  organization,  and  how  it  is 
employed  for  mission  execution.  For  JEFX  00,  the  AFRL/HEAI’s  C2TR  objective  was 
to  focus  data  collection  on  communication  and  information  flow  within  the  TCTC. 

In  order  to  employ  these  data  collection  methods  to  their  full  capacity,  access  to 
all  forms  of  communication  within  the  organization  must  be  obtained.  To  accomplish 
this,  in  previous  exercises,  the  data  collection  team  comprised  about  half  of  the 
operational  organization.  Large  numbers  of  data  collectors  are  needed  to  capture 
complete  information  flow.  The  AFRL/HEAI  assessment  team  was  limited  to  three 
people  for  Spiral  2,  Spiral  3,  and  the  Main  Experiment.  As  such,  a  decision  was  made  to 
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focus  on  “key”  leadership  positions  in  three  sections  (i.e.  Director  1  and  2  in  Hunter 
Section,  Sniper  Lead  in  Sniper  Section  and  the  MCC  in  Slayer  Section). 


The  three  observers  stationed  themselves  within  the  TCTC  and  collected 
interaction  data  focusing  on  information  flow,  communication  patterns,  teamwork 
assessment,  coordination,  and  information  management.  Data  collection,  however, 
proved  more  difficult  than  in  previous  implementations  due  to  limited  access  to  all  forms 
of  communication.  Data  collectors  had  access  to  internal  radio  communications  but 
intermittent  access  to  the  external  links  used  by  Slayer  to  conduct  battle  management. 
Monitoring  text  chat  proved  difficult  because  of  the  size  of  the  font  and  our  positioning 
next  to  or  behind  the  operators.  Overall,  the  data  collectors  were  able  to  obtain  enough 
data  to  make  a  variety  of  analytic  assessments,  but  future  data  collection  may  require 
additional  personnel  to  capture  the  complete  range  of  interactions  or  a  narrower 
collection  focus  must  be  defined. 


3.4  Sample  Data  Collection  Material 

The  C2TR  assessment  team  developed  both  paper  and  computer-based  data 
collection  tools,  however  the  computer-based  tools  were  not  allowed  into  the  compound 
due  to  security  issues.  Therefore,  only  the  paper  and  pencil  data  collection  booklets  were 
used  during  JEFX.  Originally,  to  capture  team  interactions  and  information  flow,  the 
following  Figures  (or  sheets)  were  used  during  Spiral  3.  In  Figure  3-2  the  observer  was 
to  draw  an  arrow  to  indicate  the  communication  flow  (from  section  to  section),  and  take 
copious  notes  about  the  communication  (e.g.  types  of  communications  such  as  requests 
for  information,  clarification,  etc.).  The  more  notes  that  could  be  captured  on  the 
communications  enabled  better  reconstruction  of  the  events. 

Figure  3-2  provided  observers  with  a  means  to  record  communications  patterns 
within  the  individual  TCTC  sections.  Figure  3-3  helped  observers  characterize  the  types 
of  communications  that  took  place. 
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Figure  3-2.  Layout  of  Team  to  Capture  Team 
Interactions  and  Information  Flow 
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Figure  3-3.  Questionnaire  on  the  Evolution  of  the 
Communication  Patterns  of  the  Team 


This  method  of  data  collection  proved  to  be  rather  cumbersome  as  there  were 
many  pages  to  content  with  plus  there  was  so  much  activity  audibly  and  visually  that  it 
was  a  supreme  challenge  to  try  to  keep  up  with  and  record  everything.  It  was  actually  an 
impossible  task  for  only  three  analysts  to  capture  all  that  was  occurring  in  an  operations 
cell  occupied  by  more  than  65  people  at  any  given  time. 


The  observers  also  made  use  of  the  Teamwork  Assessment  Questionnaire  Forms 
to  comment  on  the  different  aspects  of  teamwork  activities.  Figure  3-4  guided  observers 
to  assess  teamwork  issues  such  as  team  coordination,  communication,  monitoring  tasks, 
back-up  tasks,  and  team  orientation. 
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TEAMWORK  ASSESSMENT:  OBSERVER  FORM 
Communication  Tasks 

1.  To  what  extent  were  errors  caused  by  inadequate 
communication  among  cell  members? 


'  |  2  |  3  |  4  ,  5 

i  6  i  7 

1  not  at  all 

7  to  a  large  extent 

2.  To  what  extent  did  cell  members  provide  relevant 
information  to  another  cell  member,  in  a  pro-active  way, 
without  that  cell  member  having  to  ask  for  it? 

'  |  2  |  3  |  4  ,  5 

i  6  i  7 

1  never 

7  always 

Monitoring  Tasks 

3.  To  what  extent  did  cell  members  alert  each  other  to 
impending  decisions  and  actions? 

'  |  2  |  3  |  4  ,  5 

i  6  i  7 

1  not  at  all 

7  to  a  large  extent 

Back-up  Tasks 

4.  To  what  extent  did  cell  members  anticipate  the  need  to 
provide  assistance  to  other  cell  members? 

'  |  2  |  3  |  4  ,  5 

i  6  i  7 

1  never 

7  always 

5.  Did  the  cell  members  adjust  individual  task  responsibilities 
to  prevent  overload? 

I  |  2  |  3  |  4  ,  5 

i  6  i  7 

1  never 

7  always 

Coordination  Tasks 

6.  To  what  extent  was  the  cell's  behavior  coordinated? 

_ Lj _ _ Lj _ _ L 

i  6  i  7 

1  not  at  all 


7  to  a  large  extent 


Team  Orientation 

7.  How  congruent/ similar  were  the  commander  and  cell 
members’  understanding  of  the  mission? 


1  not  at  all  7  complete  agreement 


Figure  3-4.  Questionnaire  to  Help  in  the  Assessment  of  Teamwork 


Collecting  all  of  the  communications  intended  to  be  captured  on  the  collection 
form  shown  in  Figures  3-2  through  3-4  proved  to  be  more  difficult  than  expected.  As  a 
consequence,  multiple  collection  sheets  were  created  by  the  AFRL/HEA  C2TR  team 
members  throughout  the  JEFX  spiral  events  in  an  attempt  to  improve  and  optimize  the 
process  for  collecting  the  transmission  of  TCTC  intra  section  and  inter  section 
communications. 

The  inability  of  the  assessors  to  capture  all  of  the  operator  communications  was 
due  to  multiple  factors.  Limitations  in  data  capture  existed  because  of  the  inability  to 
make  use  of  the  team’s  electronic  data  collection  tools,  limited  access  to  the  operator 
communications  mediums  (headsets,  computer  processor,  or  communications  boxes),  and 
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the  restrictions  caused  by  the  team’s  reduced  numbers  and  restricted  physical  location 
within  the  TCT  Cell. 


Figures  3-5  through  3-8  are  samples  of  the  revised  collection  sheets  programmed 
for  use  during  the  Main  Experiment.  The  main  improvement  to  the  new  forms  was  the 
descriptive  depiction  of  all  the  operators  within  each  of  the  primary  sections.  This 
allowed  for  a  more  comprehensive  capturing  of  where  the  majority  of  communications 
were  taking  place  in  each  section.  The  reader  should  note  that  the  form  samples  depicted 
in  Figures  3-5  through  3-8  were  actually  printed  on  full-page  collection  sheets.  The 
sample  collection  forms  have  been  compressed  for  easy  inclusion  in  this  report. 

Located  at  the  bottom  of  each  data  collection  sheet  was  a  section  reserved  for  the 
operators  to  record  amplifying  notes  of  events  that  occurred  within  the  TCTC.  Due  to  the 
high  operations  tempo  within  the  TCTC,  it  was  quickly  discovered  that  it  was  a  very 
challenging  task  to  manually  attempt  to  record  the  vast  quantity  and  variety  of 
communications  taking  place  within  each  section.  As  a  result,  the  AFRL/HEAI  assessors 
tailored  their  data  capture  techniques  to  meet  the  requirements  dictated  by  the  operational 
nature  of  each  section  observed. 

In  addition  to  the  other  aspects  of  the  collection  forms  discussed,  at  the  very 
bottom  of  each  data  collection  sheet  was  a  key  for  identifying  the  different  forms  of 
communications  that  took  place  between  the  operators.  The  communications  key 
allowed  the  assessors  to  simply  record  a  single  letter  to  indicate  the  specific  type  of 
communications  exchange  that  occurred.  This  aided  assessors  in  the  reconstruction  of 
events  and  provided  additional  insights  into  the  manner  in  which  operators  were 
communicating. 

The  communications  key  included  the  following  symbols: 

A  =  for  use  of  the  ACCESS  comm  box  employed  for  simulation/live  communications 

B  =  for  the  Theater  Battle  Management  Core  System  (TBMCS)  Bulletin  Board  function 

C  =  for  use  of  “crib  sheets”  or  the  passing  of  written  notes  to  one  another 

D  =  for  use  of  the  Distributed  Interactive  Simulation  (DIS)  voice  communication  systems 

F  =  for  face  to  face  or  shoulder  to  shoulder  communications 

I  =  for  use  of  the  InfoWorkSpace  (IWS)  collaborative  tool 

L  =  for  use  of  the  Large  Screen  Displays  to  communicate  with  others 

N  =  for  the  Advanced  Simulation  Technology  Incorporated  (ASTI)  internal 
communications  network  box 

O  =  for  open  air,  commonly  referred  to  as  “Net  4”  or  communications  unaided  with 
electronic  devices 

P  =  for  use  of  the  common  telephone 

PA  =  for  use  of  the  public  address  system 

R  =  for  use  of  the  ground-to-air  radios  to  transmit  or  receive 
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S  =  for  use  of  the  “sneaker  net”,  i.e.,  getting  up  from  your  operator  position  to 
communicate  with  someone  else 

T  =  for  use  of  the  IWS  text  function 


The  AFRL/HEAI  recorders  were  furnished  a  headset  and  were  free  to  observe  all 
operations  and  displays.  The  Section  leadership,  personnel,  civilian  technicians,  and 
other  assessors  were  most  helpful  in  clarifying  the  information  recorded  and  offering 
explanations  and  expanding  views  to  go  with  the  information  recorded. 
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INTERCOMMUNICATION  within  TCTC 
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Figure  3-5.  Layout  of  TCTC  Team  to 
Capture  Team  Interactions  and 
Information  Flow 


Figure  3-6.  Layout  of  Hunter  Section  to 
Capture  Team  Interactions  and 
Information  Flow 


INTRACOMMUNICATION  within  Sniper  Section 


INTRACOMMUNICATION  within  Slayer  Section 
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Figure  3-7.  Layout  of  Sniper  Section  to 
Capture  Team  Interactions  and 
Information  Flow 


Figure  3-8.  Layout  of  Shotgun  Section 
to  Capture  Team  Interactions  and 
Information  Flow 
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As  mentioned  previously,  data  collection  during  the  main  experiment  proved  to  be 
somewhat  difficult.  This  was  especially  true  in  situations  where  the  assessor  was  without  access 
to  the  operator’s  communication  equipment.  As  a  result,  impromptu  methods  were  used  to 
capture  as  much  of  the  activities  occurring  as  possible.  Many  times  the  observers  just  quickly 
wrote  down  the  communication  chronologically  in  a  log  format,  specifying  the  time  for  each 
communication,  how  the  message  was  sent,  who  sent  the  message,  and  who  received  the 
message.  Figure  3-9  highlights  the  complexity  of  capturing  the  myriad  of  communication 
sources  and  avenues  that  were  in  use  during  JEFX  00. 
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Figure  3-9.  Example  Predator/UAV  Workstation  Connectivity 


Each  communication  was  noted  as  either  coming  into  the  player  position  where  the 
observer  was  collocated,  or  going  out.  It  should  be  noted  that  the  communications  captured  for 
other  player  positions,  (i.e.  Killer  Lead)  were  only  from  the  perspective  of  the  player  position  in 
which  the  observer  was  collocated.  The  communications  can  really  only  be  assessed  with  an 
observer  collocated  at  the  position  in  interest. 

The  communications  were  categorized  in  two  ways.  One  categorization  was  the  mode  of 
communication  used  by  the  TCTC  operator  such  as  face-to-face,  internal  simulation  networks,  or 
UHF  radio  methods.  Another  categorization  was  the  type  of  communication  that  took  between 
the  TCTC  operators.  The  types  of  categorizations  used  to  describe  the  types  of  communications 
between  the  TCTC  operators  include  the  following:  (1)  Requests  for  information  (RFI);  (2) 
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Coordination  communications;  (3)  Directive  Orders,  (4)  Information  updates,  or  (5) 
Communication  responses  or  confirmations 


In  the  course  of  the  spiral  nature  for  improving  the  data  capture  methods,  another  form 
was  developed  to  capture  in  an  organized  manner  as  much  of  the  activities  occurring  as  possible. 
From  the  experience  gained  during  JEFX  00  Spirals  2  and  3,  it  became  clear  that  the  data  capture 
process  would  require  a  form  that  was  simple  in  design,  yet  formatted  in  such  a  way  as  to  allow 
the  comprehensive  logging  of  desired  information.  Table  3-1  is  an  example  of  the  form  used. 


Table  3-1.  Example  Communications  Transmission  Capture  Form 
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As  a  result  of  the  plethora  communications  captured  and  the  observation  of  all  TCTC 
activities,  data  was  captured  that  allowed  analysts  to  produce  workload/targetload  charts 
analogous  to  the  one  depicted  in  Figure  3-10.  This  type  of  data  is  useful  for  tracking  all  TCTC 
target  category  lists,  e.g.  Emerging  Targets  and  Designated  Targets.  This  could  be  used  as  a 
barometer  for  the  human  operator  work/stress  levels  and  overall  TCTC  activity.  It  could  be  a 
useful  leadership  and  manpower  utilization  management  tool  and  could  potentially  be  projected 
onto  the  Large  Screen  Display  during  TCTC  operations. 


Figure  3-10.  Sample  TCTC  Target  Load  Over  Time 

3.5  Approach  to  the  Team  Leadership  Assessment 

It  was  predicted  early  in  the  assessment  planning  process  that  leadership,  something  that 
no  previous  JELX  had  addressed,  would  be  a  key  factor  in  the  successful  operation  of  the 
experimental  TCT  Cell.  This  hypothesis  was  the  foundation  of  this  analysis  and  was  found  to  be 
absolutely  true.  This  section  describes  the  circumstances  and  setting  for  the  experiment,  the 
measures  of  merit  and  the  results  of  watching,  listening  and  gather  evidence.  It  was  found  that 
the  individuals  chosen  as  TCT  Cell  leaders  provided  the  interpersonal,  conceptual,  technical  and 
tactical  skills  needed  to  focus  team  purpose,  provide  coordinated  direction  and  produce  overall 
group  motivation  to  accomplish  the  mission.  In  all,  the  TCTC  leadership’s  displayed  all  the  Air 
Lorce  values  and  demonstrated  warfighting  staff  planning  competencies. 

ALRL/HEAI  analysts  devised  an  assessment  plan  to  identify  and  report  on  the 
forms/demonstrations  of  leadership  which  were  necessary  to  create  a  successful  and  functioning, 
environment  for  accomplishing  the  time  critical  targeting  cell  mission  from  an  eclectic  group  of 
commanders,  operators,  staff  officers,  mission  planners  and  highly  technical  communications, 
electronic,  and  intelligence  specialists.  By  all  standards,  this  organization  was  a  “work-in- 
progress”  operation  of  individuals  learning  to  create  operational  procedures  and  using  newly 
acquired  and  previously  untried  equipment.  The  goal  of  the  assessment  was  to  identify  the 
leadership  requirement  needed  to  create  a  team  to  detect,  decide,  devise,  and  destroy  targets. 

3.6  Design  of  Leadership  Assessment 

In  order  to  efficiently  capture  information  about  leadership,  it  was  necessary  to 
characterize  military  leadership  into  a  “schema”  for  leadership  identification.  Next,  because  of 
the  number  of  individuals  involved  in  the  TCTC,  it  became  necessary  to  identify  the  particular 
individuals  to  observe  exercising  leadership.  Lor  collection  purposes  it  was  also  necessary  to 
create  the  data  tables  needed  to  record  the  key  information  which  captured  the  instances  where 
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this  leadership  was  being  demonstrated.  These  tasks  were  begun  by  identifying  the  areas  that 
represented  leadership  using  the  information  derived  from  multi-service  military  references  and 
next  by  identifying  those  positions  within  the  TCTC  for  which  responsibilities  were  assigned  or 
delegated.  The  next  part  of  the  task  was  to  note  the  environment  and  circumstances,  and  to 
collect,  in  written  form,  those  demonstrated  attitudes,  deliberate  and  aside  statements,  and  most 
importantly  the  actions  which  guided  the  behavior  or  opinion  of  others  and  directed  their 
performance  or  activities  toward  achieving  certain  goals  or  results.  With  these  collections  of 
attitude,  words,  and  actions  the  next  step  would  be  the  proper  categorizing  and  reviewing  of  the 
information  for  analysis  and  reporting. 

3. 7  Measures  of  Merit 

Leadership  was  looked  at  from  a  design  matrix  of  four  dimensions  (values,  attributes,  skills,  and 
actions)  in  17  leader- specific  categories.  We  sought  to  define  each  of  the  categories  and  to 
create  a  means  to  identify  instances  of  demonstrated  leadership  by  individual’s  comments,  by 
their  representations  of  operational  knowledge,  and  by  the  manner  in  which  they  took  action  to 
motivate  and  lead  their  parts  of  the  organization.  Table  3-2  represents  the  schema  used  for 
looking  at  leadership  and  formed  the  measures  of  merit  in  judging  examples  to  validate  the 
hypothesis. 


Table  3-2.  Leadership  Categories 


JEFX  00  Leadership  Schema 
(Four  Dimensions) 

Values  (7) 

Attributes  (3) 

Skills  (4) 

Actions  (3) 

C01  Loyalty 

C08  Mental 

Cll  Interpersonal 

C15  Influencing 

C02  Duty 

C09  Physical 

C12  Conceptual 

C16  Operating 

C03  Respect 

CIO  Emotional 

C13  Technical 

C17  Improving 

C04  Self-service 

C14  Tactical 

C05  Honor 

C06  Integrity 

C07  Personal  courage 

3.8  Identification  of  the  Specific  Leaders  to  Observe 

Although  the  TCT  Cell  contained  many  leaders  and  leadership  examples  at  every  level, 
the  following  chart  indicates  those  leadership  positions  that  received  primary  attention.  It  is 
important  to  note  that  the  Hunter  Section  had  a  director  lead,  a  target  lead  and  a  fusion  lead,  and 
the  Killer  Section  had  a  mission  planner  lead  (Sniper),  mission  control  lead  (Slayer)  and  a 
weapons  director  (weapons  portion  of  the  Slayer  team).  Figure  3-11  outlines  the  positions 
observed  in  the  leadership  assessment. 
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Figure  3-11.  TCTC  Principal  Leadership  Schema 


A  very  specific  rationale  was  developed  for  the  selection  and  comments  about  each  of  the 
positions  identified.  In  all  cases,  the  comments  collected  were  made  by  individuals  who  will  not 
be  identified  by  name  or  gender  in  this  report.  AFRL/HEAI  is  prohibited  from  reporting  any 
JEFX  00  results  until  the  AFEO  staff  has  out-briefed  the  Chief  of  Staff  of  the  Air  Force.  JEFX 
00  was  an  experiment,  which  shows  only  feasibility.  It  is  important  for  the  reader  to  take  note  of 
the  fact  that  assessment  procedures  must  be  examined  in  a  laboratory  environment,  which  takes 
into  consideration  all  the  variables  that  were  not  controlled  in  order  to  show  improvement  and 
establish  causality. 


SECTION  4.0 
SUMMARY 


4.1  Summary  Thoughts 

The  human  operator  is  a  critical  link  in  the  effective  functioning  of  command  and  control 
operation  centers.  Modern  C2  centers  have  incorporated  ever  increasing  advances  in  warfighting 
technology  that  spans  the  man-machine-interface  continuum  of  operator  displays,  decision  aides 
computing  systems,  precision  location  processes,  and  complex  and  sophisticated  data  link 
communications.  JEFX  00  provided  AFRL/HEAI  analysts  with  an  excellent  environment  in 
which  to  gain  insight  into  the  implications  these  modern  technological  advances  and  changing 
concepts  of  operations  pose  for  the  human  element. 
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Under  the  circumstances  afforded  the  AFRL/HEAI  analyst,  considerable  amounts  of  data 
were  collected  to  allow  for  substantive  assessments  to  be  made  of  the  many  factors  that  can 
effect  the  functioning  of  the  modern  C2  warfighter.  Although  the  results  of  the  AFRL/HEAI 
assessments  cannot  be  publicly  shared  at  this  time,  valuable  information  was,  in  fact,  derived 
from  JEFX  00  that  begins  to  help  characterize  C2  operator  workloads,  cognitive  loads,  individual 
and  team  processes,  critical  information  flow,  decision  making,  communication  patterns,  and  the 
important  dynamics  and  impact  of  leadership. 

4.2  Significance  of  Leadership 

The  nature  of  experimentation  is  quite  different  from  that  of  a  traditional  Air  Force 
exercise  or  contingency  operation  and  JEFX  00  included  many  frustrations  and  challenges  that 
tested  the  professionalism  and  endurance  of  the  TCTC  personnel.  Strong,  “hands-on”  leadership 
was  needed  to  maintain  the  intensity  of  the  operations  and  keep  the  operators  focused  - 
especially  during  the  idiosyncrasies  of  distributed  simulation  and  the  tightly  controlled  peacetime 
live-flying  operations. 

Leadership  was  the  most  important  factor  in  the  success  of  the  TCT  Cell.  Generally 
speaking  the  TCTC  leaders  scored  high  in  all  17  leadership  categories  and  affected  every  aspect 
of  the  TCTC  operation.  They  provided  the  interpersonal,  conceptual,  technical,  and  tactical 
skills  needed  to  focus  the  team,  provided  coordinated  direction,  and  produced  overall  group 
motivation  to  accomplish  the  mission. 

Because  the  TCTC  decision-makers  were  teamed  directly  with  the  supporting 
information  providers,  the  collective  TCTC  organization  was  able  to  effectively  accomplish  the 
C2  decision  making  process  for  time-critical  targeting.  In  addition,  the  assessment  team  was 
able  to  devise  and  capture,  for  analysis,  the  activities  and  efforts  required  to  connect  the  “sensor- 
to-shooter”  operating  mechanisms  and  to  display  this  functionally  using  descriptive  measures 
useful  to  the  warfighter. 

Analysis  of  the  communications  network  was  the  key  to  assessing  the  organization,  its 
structure,  the  personnel  and  system  requirements,  and  the  training  needed  to  measure  and 
accomplish  the  mission.  The  communications  network  was  the  circulatory  design  of  the 
operation  and  the  information  was  the  pulse.  Measurements  of  the  communications  network  and 
its  operation  showed  it  to  be  more  extensive  and  more  critical  than  previously  imagined.  The 
communications  nets  contained  all  forms  of  voice,  text-based,  imagery,  and  electronic 
information  and  were  supplemented  by  direct  voice  communications  and  face-to-face 
discussions.  The  data  contained  in  this  report  reflects  the  extensive  communications  that  took 
place  in  order  to  prosecute  time- sensitive  targets. 

The  outcome  of  the  analysis  of  the  operator  communications  supported  the  TCTC 
leadership’s  direction  to  constantly  push  information  up  stream.  The  full  cycle  of  battle 
management  was  supported  in  a  single  location,  which  facilitated  an  iterative  feedback  loop 
within  the  TCTC  organization.  Conversations  on  the  internal  links  between  the  various  section 
leads  resulted  in  a  better  decision  making  process  and,  in-turn,  better  weapon  solutions.  The 
resident  expertise  on  targets,  weapons,  and  assets  was  distributed  across  the  organization  and  the 
exchange  of  this  critical  data  served  to  integrate  knowledge  and  improve  overall  TCT  process 
outcomes. 
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The  process  of  capturing  this  assessment  information,  along  with  further  analysis  of  the 
data  collected,  makes  it  possible  to  examine  the  TCTC  operation  in  a  warfighter  laboratory 
environment  where  processes  and  systems  can  be  closely  examined  and  optimized.  The 
information  contained  in  the  AFRL/HEAI’s  “JEFX  2000  Time  Critical  Targeting  Cell  Process 
Initiative  Report”  contains  a  lot  of  observed  data  that  will  be  useful  to  the  warfighter,  the  C2 
operational  community,  and  will  provide  the  basis  for  further  research  and  designs  for  required 
C2  assessment  and  training. 


The  TCTC  Process  Initiative,  as  executed  during  JEFX  00,  should  be  comprehensively 
analyzed  and  further  refined.  There  is  a  clear  military  necessity  for  having  a  well-trained  cadre 
of  personnel  capable  of  dynamically  and  efficiently  responding  to  time  sensitive  and  critical 
targets.  In  order  to  meet  this  goal  with  the  correct  type  and  mixture  of  human  operators,  a  more 
detailed  analysis  of  the  micro  and  macro  processes,  communication  patterns,  and  team 
interactions  required  within  the  TCTC  needs  to  occur. 
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